Decoding machine learning benchmarks

Despite the availability of benchmark machine learning (ML) repositories (e.g., UCI, OpenML), there is no standard evaluation strategy yet capable of pointing out which is the best set of datasets to serve as gold standard to test different ML algorithms. In recent studies, Item Response Theory (IRT) has emerged as a new approach to elucidate what should be a good ML benchmark. This work applied IRT to explore the well-known OpenML-CC18 benchmark to identify how suitable it is on the evaluation of classifiers. Several classifiers ranging from classical to ensembles ones were evaluated using IRT models, which could simultaneously estimate dataset difficulty and classifiers' ability. The Glicko-2 rating system was applied on the top of IRT to summarize the innate ability and aptitude of classifiers. It was observed that not all datasets from OpenML-CC18 are really useful to evaluate classifiers. Most datasets evaluated in this work (84%) contain easy instances in general (e.g., around 10% of difficult instances only). Also, 80% of the instances in half of this benchmark are very discriminating ones, which can be of great use for pairwise algorithm comparison, but not useful to push classifiers abilities. This paper presents this new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks.Comment: Paper published at the BRACIS 2020 conference, 15 pages, 4 figure

Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox

This article discusses three statistical paradoxes that pervade epidemiological research: Simpson's paradox, Lord's paradox, and suppression. These paradoxes have important implications for the interpretation of evidence from observational studies. This article uses hypothetical scenarios to illustrate how the three paradoxes are different manifestations of one phenomenon – the reversal paradox – depending on whether the outcome and explanatory variables are categorical, continuous or a combination of both; this renders the issues and remedies for any one to be similar for all three. Although the three statistical paradoxes occur in different types of variables, they share the same characteristic: the association between two variables can be reversed, diminished, or enhanced when another variable is statistically controlled for. Understanding the concepts and theory behind these paradoxes provides insights into some controversial or contradictory research findings. These paradoxes show that prior knowledge and underlying causal theory play an important role in the statistical modelling of epidemiological data, where incorrect use of statistical models might produce consistent, replicable, yet erroneous results

Radiogenomic analysis of primary breast cancer reveals [18F]-fluorodeoxglucose dynamic flux-constants are positively associated with immune pathways and outperform static uptake measures in associating with glucose metabolism

Background: PET imaging of 18F-fluorodeoxygucose (FDG) is used widely for tumour staging and assessment of treatment response, but the biology associated with FDG uptake is still not fully elucidated. We therefore carried out gene set enrichment analyses (GSEA) of RNA sequencing data to find KEGG pathways associated with FDG uptake in primary breast cancers. Methods: Pre-treatment data were analysed from a window-of-opportunity study in which 30 patients underwent static and dynamic FDG-PET and tumour biopsy. Kinetic models were fitted to dynamic images, and GSEA was performed for enrichment scores reflecting Pearson and Spearman coefficients of correlations between gene expression and imaging. Results: A total of 38 pathways were associated with kinetic model flux-constants or static measures of FDG uptake, all positively. The associated pathways included glycolysis/gluconeogenesis (‘GLYC-GLUC’) which mediates FDG uptake and was associated with model flux-constants but not with static uptake measures, and 28 pathways related to immune-response or inflammation. More pathways, 32, were associated with the flux-constant K of the simple Patlak model than with any other imaging index. Numbers of pathways categorised as being associated with individual micro-parameters of the kinetic models were substantially fewer than numbers associated with flux-constants, and lay around levels expected by chance. Conclusions: In pre-treatment images GLYC-GLUC was associated with FDG kinetic flux-constants including Patlak K, but not with static uptake measures. Immune-related pathways were associated with flux-constants and static uptake. Patlak K was associated with more pathways than were the flux-constants of more complex kinetic models. On the basis of these results Patlak analysis of dynamic FDG-PET scans is advantageous, compared to other kinetic analyses or static imaging, in studies seeking to infer tumour-to-tumour differences in biology from differences in imaging. Trial registration NCT01266486, December 24th 2010

ERCC1 expression and RAD51B activity correlate with cell cycle response to platinum drug treatment not DNA repair

Background: The H69CIS200 and H69OX400 cell lines are novel models of low-level platinum-drug resistance. Resistance was not associated with increased cellular glutathione or decreased accumulation of platinum, rather the resistant cell lines have a cell cycle alteration allowing them to rapidly proliferate post drug treatment. Results: A decrease in ERCC1 protein expression and an increase in RAD51B foci activity was observed in association with the platinum induced cell cycle arrest but these changes did not correlate with resistance or altered DNA repair capacity. The H69 cells and resistant cell lines have a p53 mutation and consequently decrease expression of p21 in response to platinum drug treatment, promoting progression of the cell cycle instead of increasing p21 to maintain the arrest. Conclusion: Decreased ERCC1 protein and increased RAD51B foci may in part be mediating the maintenance of the cell cycle arrest in the sensitive cells. Resistance in the H69CIS200 and H69OX400 cells may therefore involve the regulation of ERCC1 and RAD51B independent of their roles in DNA repair. The novel mechanism of platinum resistance in the H69CIS200 and H69OX400 cells demonstrates the multifactorial nature of platinum resistance which can occur independently of alterations in DNA repair capacity and changes in ERCC1

Answer changing in multiple choice assessment change that answer when in doubt – and spread the word!

<p>Abstract</p> <p>Background</p> <p>Several studies during the last decades have shown that answer changing in multiple choice examinations is generally beneficial for examinees. In spite of this the common misbelief still prevails that answer changing in multiple choice examinations results in an increased number of wrong answers rather than an improved score. One suggested consequence of newer studies is that examinees should be informed about this misbelief in the hope that this prejudice might be eradicated. This study aims to confirm data from previous studies about the benefits of answer changing as well as pursue the question of whether students informed about the said advantageous effects of answer changing would indeed follow this advice and change significantly more answers. Furthermore a look is cast on how the overall examination performance and mean point increase of these students is affected.</p> <p>Methods</p> <p>The answer sheets to the end of term exams of 79 3^rdyear medical students at the University of Munich were analysed to confirm the benefits of answer changing. Students taking the test were randomized into two groups. Prior to taking the test 41 students were informed about the benefits of changing answers after careful reconsideration while 38 students did not receive such information. Both groups were instructed to mark all answer changes made during the test.</p> <p>Results</p> <p>Answer changes were predominantly from wrong to right in full accordance with existing literature resources. It was shown that students who had been informed about the benefits of answer changing when in doubt changed answers significantly more often than students who had not been informed. Though students instructed on the benefits of changing answers scored higher in their exams than those not instructed, the difference in point increase was not significant.</p> <p>Conclusion</p> <p>Students should be informed about the benefits of changing initial answers to multiple choice questions once when in reasonable doubt about these answers. Furthermore, reconsidering answers should be encouraged as students will heed the advice and change more answers than students not so instructed.</p